ON THE VARIETY OF ALMOST COMMUTING NILPOTENT 

MATRICES 



ELIANA ZOQUE 



Abstract. We study the variety of n x n matrices with commutator of rank at 
most one. We describe its irreducible components; two of them correspond to 
the pairs of commuting matrices, and n — 2 components of smaller dimension 
corresponding to the pairs of rank one commutator. In our proof we define a 
map to the zero fiber of the Hilbert scheme of points and study the image and 
the fibers. 



1. Introduction 

Let be a vector space of dimension n over a field K of characteristic equal to or 
> n/2. Let g — gl^{V) and n be the nilcone of g, i.e., the cone of nilpotent matrices 
of g. We write elements of V and V* as column and row vectors, respectively. In 
this paper we study the variety 

TV ;= {{X,Y,i,j) enxnxVxV*\ [X,Y]+ij = 0} 

and prove that it has n irreducible components: 2 of dimension n'^ + n — 1 corre- 
sponding to the case where the matrices commute, and n — 2 of dimension n^+n~2, 
corresponding to the noncommutative pairs. 

The pairs of almost commuting matrices have been studied recently in [7] , where 
Gan and Ginzburg study the structure of the scheme 

M := €2xgxV xV*\[X,Y]+ij ^ 0}. 

They prove that the irreducible components of A4 are the closures of the sets 
A^o, Ml, . . . , Mn, defined as 

A4t = {{X,Y,i, j) G A4\Y has pairwise distinct eigenvalues and 

dimK{X, Y}i = t, dimjK(X,Y) = n - t} 

where K{X, Y)i (resp. jK{X, Y)) is the smallest subspace oiV (resp. V*) containing 
i (resp. j) and invariant under X and Y. 

Let JCn = {{X,Y) G n X n| [X, y] = 0}, the variety of commuting nilpotent 
matrices. Baranovsky proved in [T] that /C„ is irreducible and has dimension — 1. 
In his proof, he shows that U = {{X, Y, i) e JCn x V \ K[X, Y]i ~ V} is irreducible, 
dense in /C„ x F an has dimension + n — 1. GL{V) acts faithfully on U, and the 
quotient U/GL{V) is a fiber of the Hilbert scheme of points under the Hilbert-Chow 
morphism. 

Let 

Afr^s = {{X,Y,iJ) eJ\f\ dimK{X,Y)i = r, dimjK{X,Y) = s}, 
AC,s = {iX,Y,i,j) eAf\ diinK{X,Y)i < r, dimjK{X,Y) < s} 
and A7"r,s the Zariski closure of A/'r.s- Clearly A/'r,s C 77r,s ^ A/^^s- 
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Since X and Y can be put in upper triangular form simultaneously ([6], Lemma 
12.7), we have that [X, Y] is not only strictly upper triangular, but the entries that 
are located two positions above the diagonal are zero too. Therefore, 

A/' = 7Vo,„ U7V„,o U y Ur,s. 

0<r+s<n 

Following Baranovsky, we have that ATo^n = TVq „ and Unfi — o which can 
be identified with /C„ x V and /C„ xV* , respectively. 
Our main theorem is the following. 

Theorem 1. (a) The irreducible components oj M are precisely 7\ft^n-i-ti 1 ^ 
t<n~~2, A^(5 „ and N^q. 
(b) diva J\ft.n-i-t — + n — 2 for I < t < n — 2 and dimA/'o „ — dimA/"^ g = 
71^ + n — 1 . 

In the last section we study the variety 

S = {{A,B,i,j) (^-axxixV xV*\A + B = ij}. 

and describe its irreducible components. The key fact is that if {A, B, i, j) £ S then 
A and B are simultaneously triangularizable. The considerations that we make for 
this variety are simpler that the ones for Af since in this case it is easy to deform an 
element of S and stay inside S, one just has to consider matrices that are strictly 
upper triangular in the given basis. 

2. The Hilbert scheme 

In this section we establish a connection between the Hilbert scheme and Aft,n-i-t 
to prove that the later is irreducible if charK = or > n/2. This will also allow us 
to prove part (b) of Theorem [TJ 

Clearly every element of K{X,Y)i (resp. jK{X,Y)) can be written as p{X,Y)i 
(resp. jp{X,Y)) where p{x,y) G ¥^{x,y) is a polynomial in the noncommutative 
variables x and y. 

Lemma 2. j¥L{X,Y) (resp. M.{X,Y)i) is a right (resp. left) ¥i[x,y]-module and 
its perpendicular complement {jK{X,Y))-^ = {v E V \uv — ^/u ^ jK{X,Y)} 
(resp. (K{X,Y)i)^ = {2 e | = Vw G K{X,Y)i}) is a left (resp. right) 
K[x,y]-module, where x and y act as X and Y respectively. 

Proof. jK{X,Y) and {jK{X,Y))-^ are right and left IK(a;, ?;)-modules, respectively. 
We have to prove that the two-sided ideal generated by [x, y] acts as zero on both 
of them. 

Let qi{x,y), q2{x,y) G K{x,y). Then for every v G {jK{X,Y))^ , 

qi{X, Y)iXY - YX)q2iX, Y)v = qiiX, Y)i{jq2{X, Y)v) = 

since jq2{X,Y) G K{X,Y). This proves that (jK{X,Y))^ is a K[a;, ?;]-module 

To prove that jK{X, Y) itself is a K[x, ?;]-module, note that jK{X, Y) C {{X, Y)iy 
since the elements of jR{X, Y) and K{X, Y)i have the form jp{X, Y), q{X, Y)i for 
pix,y), q{x,y) G K(a;,y), and jp{X,Y)q{X,Y)i = ([7], Lemma 2.1.3.). This 
proves the claim. □ 



From now on, we write jK[X, Y] = jK(X, Y), K[X, Y]i = K{X, Y)i. 
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We now define a map Mr,s W[o] x Wj'o] "^here Ti'^j is the fiber of the Hilbert- 
Chow morphism W^ih?) S"'{A^) over the point m ■ [0] e 5™(A2) and n"^{A^) 
denotes the Hilbert scheme of points in the affine plane. The image of {X, Y, i, j) € 
Mr, 8 to Hjgj is the ideal {p[x,y) S IK[a:,y] \ p{X,Y)i — 0}. This ideal is also equal 
to {p{x,y) G K[a;,y] \ p{X,Y)v = Vz; G K[X, Fji} since j is a cychc vector for 
X and Y on K[X, Similarly, we define Mr.s "H^q] as the ideal {p{x,y) € 
K[x,y]\jpiX,Y)=0}. 

Since (K[X, yjz)-*- and (j]K[X, F])-'- are K[a;, j/]-modules, one could try to induce 
maps A/'r.s ■'^[0]'*' ^[of'^' there may not be a cyclic vector for the actions of 
X and Y on those spaces. However, if r + s = n — 1 the following theorem implies 
that such map does exist. 

Theorem 3. There is a well-defined regular map Mt,n-i-t — > induced by 

the actions of X and Y on {jK.[X, Y])-^ . This map is dominant and the image of any 
element of Mt,n-i-t has the form {y^'^^ , x—aiy—- ■ ■—aty*') or (x*, y—aix—- ■ -—atx*) 
for some ai , . . . , at G K. 

Similar considerations hold for the map M,«-i-t H"o7*(A2) induced by the 
actions of X and Y on (K[X, In order to prove Theorem [1] we study the 

image and the fibers of the map ^E* : Nt,n-i-t '^[a] ^ ■'^[of*' Recall that GL„(K) 
acts on AAby G- = {GXG-^ ,GYG-^ ,Gi, jG-^). 

Theorem 4. The fibers of the map * : 7V(,n-i-t ^ 'Wjq] x'WjJ,]"^"* are the GL„(K)- 
orhits, and the isotropy of each element of Aft,n-i-t is one- dimensional. 

In order to prove Theorems [3] and |4] we need a technical result. 

Lemma 5. Let (X,Y,i,j) G JVr,s- There is a basis {ei, . . .e„} ofV with dual basis 
{e*, . . . e* } ofV* so that = i, ^n+i-s ~ J' ^'"^'^ ^ '^'^^ upper triangular in this 
basis. 

Proof. Since j]K[X, Y] annihilates K[X, Y]i we can decompose y = 14 © V2 © V3 so 
that Vi = K[X, Y]i and = (jK[X, Y])* . We are to find elements ei, . . . , G 
Vl, e^+i, . . . , e„_3 G V2, Cn+i-s, . . . , e„ G V3 satisfying the conditions. 
Consider the lex deg order of the monomials in K[.'e, y] 

1 < X < y < x^ < xy < y'^ < x^ < x^y < xy'^ < y^ < . . . 

Choose the largest (according to <) monomial mi so that mi{X, Y)i ^ (it exists 
since X°'Y^i = if a+6 > n), and inductively choose as the largest monomial so 
that mk(X,Y)i is not a linear combination of mi{X,Y)i, • • • , m^._i{X,Y)i. This 
gives us r monomials mi, • • • , m^ so that K[X, Y]i = {mi{X, Y)i, • • • , mr{X, Y)i). 
We set ei = mi{X, Y)i, . . . , = mr{X, Y)i — i. The action of X and Y in this 
basis is triangular, since multiplying by x or ?/ "increases" monomials, and for every 
monomial m, either m{X, Y)i is in the basis or is a linear combination of larger 
monomials. That e,- = i is a consequence of the following lemma which will also be 
useful later. 

Lemma 6. Let mt{x, y) — x°-*y^* and A = {(oi, . . . , (a^, fer)}- If (a, 6) G A and 
< a' < a, < b' < b then (a', b') G A. 
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Proof. If (a', b') ^ A then X° Y'^ i can be written as a linear combination of larger 
monomials ^ Q;c,ci-'^'^i^''i- Then 



mt{X,Y)i = X""" y 



6-h' 



^j.a yb' <^x^y'^ 

but this is a contradiction since < x^y^ implies x'^y^ < yd+b-b _ |— | 



2;a yfa <;;2;Cyd 



In particular this implies that mr{x,y) — 1 and therefore = i. 

Now we can follow the same procedure in V3* = jK[X, Y] to find elements 



, e* e jIK[X, F] which in turn give rise to e„+i_ 



e„ e C V. 



'^n+l — SJ ■ 

To find the remaining elements of the basis, note that since V2 C (jK[X, F])^ 
we have that jK[X,Y]XV2, jK[X,Y]YV2 C jK[X,Y]V2 = 0. This means that 
X\v2,Y\v2 ■■ V2 ^ Vi ® ¥2- Since X,Y : Vi Vi, the actions of X and F as 
endomorphisms of V2 are nilpotent, and they are commutative since [X, Y] = ij 
which acts as on this space. Therefore we can find e^+i, . . . , e„_s G V2 that make 
both X and Y upper triangular when restricted to V2. The basis {ei, . . . e„} of 
satisfies the conditions. □ 

Therefore we can assume that X and Y are upper triangular, 






1 






, K[X,Y]i = 






Vo/ 



,j= 



1 



0) 



jK[X,Y]^(Q 







where the I's in i and j are located in the r-th and (71 + 1 — s)-th position, respec- 
tively. 

Proof of Theorem\^ We have to prove that (jK[X, Y\)^ admits a cyclic vector and 
that one of X, Y is regular when restricted to that space. 

Let w G {jK[X,Y])^ \K[X,Y]i. Since Xw.Yw G {jK[X,Y])^ = Kw 
K[X, y]i and X^Y are nilpotent, we have that Xw^Yw G K[X, Fji; let Xw — 
P[X, Y)i, Yw — Q{X, Y)i, where P and Q are polynomials. Let Q{x, y) — Qi{x, y)+ 
c where Qi(0, 0) = 0. If c 7^ then we have 

-{YP{X, Y) ~ XQi{X, Y))i = -{YP{X, Y) - XQ{X, Y) + cX)% 
c c 

= i(yX - XY)w + Xi= -ijw + Xi = Xi 
c c 

and every monomial in the left-hand side is > a; in the lex deg order. Multiplying 
by X°'~^Y'', a > 0, 6 > 0; we conclude that X°-Y^i is a linear combination of larger 
monomials. According to the construction of the basis in Lemma [5] we have that 
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the basis for K[X, y]i is {Y'^i, . . . , Yi, i} and this impHes that Y acts regularly in 
K[X,Y]i. 

If P has a constant term we can reverse the roles of a;, y in the lex deg order. 
Now we prove that at least one of the polynomials P, Q has a nonzero constant 
term. 

We can choose u €V \ (jK[X,Y])^ so that Xu,Yu G {jK[X,Y])^: to do this 
take any uq G V \ {jK[X,Y])-^; if Xuo,Yuo G {jK[X,Y])-^, take u = uq. If not, 
say Xuq ^ {jK[X,Y])^ , take ui — Xuq and repeat the process. 

Then ju ^ and Xu ^ aw + R{X, Y)i, Yu = f3w + S{X, Y)i for some R,S G 
K[x,y], a,/3 G K. If jw = then jK[X,Y]u = since Xu.Yu G {jK[X,Y])^. We 
can assume, normalizing u if necessary, that ju = 1 . Therefore 

i = iju = XYu - YXu = X{Pw + Six, Y)i) - Y{aw + R{X, Y)i) 

= (/3P(X, Y) - aQ{X, Y) + XSiX, Y) - YR{X, Y))i. 

If P and Q have no constant term then all the monomials in the right-hand side 
have positive degree, a contradiction. 

Therefore one of X or F acts regularly on K[X, F]. Assume without loss of 
generality that it is Y. It is easy to see that then X|K[x,i']i = ^(^)|K[x,y]i where 
A{y) = ai + • • ■ + at-iy'-^ G IK[y] 

Now we prove that there exists i' G Y])-^ \K[X, Y]i so that Yi' — i. Since 

Xw, Yw G K[X, Y]i = K[Y]i, let 

Xw = D{Y)i ^ {do+diY+- ■ ■+dt-iY'~^i), Yw = C(F)i = {co+ciY+- ■ ■+ct-iY''^)i. 

If Co ^ let i' = ^(w -cii ct-iY*~^i) G Kw © K[Y]i = {jK[X,Y])^. So 

assume by contradiction that cq = 0. Then jw = implies 

= ijw = {XY - YX)w = {A{Y)C{Y) - YD{Y))i 

and therefore, comparing the coefficient of Y, do = 0- 
Let u € V a,s before. Then 

i = iju = XYu - YXu = X{(3w + S{X, Y)i) - Y{aw + R{X, Y)i) 

= {I3D{Y) - aC{Y) + XS{X, Y) - YR{X, Y))i; 

but all the monomials on the right-hand side have positive degree, which is impos- 
sible. Therefore we can find such i' . 

But Xi' = ^{Xw - ciXi Ct-iXY*-^i) G K[X,Y]i =C {jK[X,Y])^. 

Therefore K[X,Y]i C K[X,Y]i' C {jK[X,Y])^. This and dim(jIK[X, F])^ = 1 + 
dimK[X,Y]i imply that {jK[X,Y])^ = K[X,Y]i' = K[Y]i' which means that i' is 
a cyclic vector in F])^ and therefore we have a map Aft.n-i-t — > 'H*^o]^{A'^). 
If Xi' = {aiY + a2Y^ + ■■■ + atY')i' then the image of {X,Y,i,j) in Wf^y (A) is 
{y^+\ X - aiy - a2y^ aty*). □ 

Proof of Theorem^ Let Xr.s and y^.s denote the entries in the r-th row and s-th 
column of the matrices that represent X and Y respectively in the basis described 
in Lemma O 



Since 



7^ we can assume without loss of generality that 



2^4-1-1,* yt+i,t 
xt+2,t+i yt+2,t+i 

xt+i,t,yt+2,t+i 0- Therefore 

K[X,Y]t^{i,Yi,...,Y'-'),jK[X,Y] = {j,jX,...,jX''-'-') 
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Consider the filtration Wa ^ (jIK[X, Y])^ = {jJX, . . .,jX"-^-*)^, Wi ^ 
(jX, . . . , . . . , T4^„_i_t = 

We choose a basis {vi, . . . , vt,vt+i, . . . , «„} of V so that vi = F* ^i, . . .Vt = 
i, vt+i =i' &Wo\ K[X, Y]i, vt+2 eWi\Wo,. ■ ■ , Vn eV\ Wn-i-t, and we can do 
this in such a way that 











q ' 


> 


















Xp^q — < 


ai 


q - 
q - 


-p = 
-p = 


1, 1 < P < i 
l,t+l<p< 


n — 


1 


















q - 




1 < p < n, t 


+ 


i<q 


< n 














q - 


-p>0 


















Vp, 


,= < 




q - 
q - 


-p = 
-p = 


1, 1 < P < t 
1, t + l<p< 


n — 


1 


















q - 


-p^l, 1 <p <t + 


l,l<q<t 


+ 1 






where \I>(X,y,i,j) = 


, {x 


-oiy 






-bix — - 




K-i- 


-tx""-^ 


and aibi 




























For example, for n ; 




4: 




















/O ai 


0.2 


as 




xie 


Xl7\ 






1 








2/16 


Vn\ 







ai 




03 


X2e 


X27 










1 





2/26 


2/27 










ai 


02 


X36 


X37 










1 





2/36 


2/37 


X = 











Ol 


X46 


X47 


, Y = 











1 


2/46 


2/47 
















1 



















hi 


h2 



















1 



















bi 




[o 














J 




^0 














0; 



I = 






1 










1 





,j=(0 ai6i-l 0) . 



(Note that we are changing j from our previous notation, this is to simplify our 
expressions for X and Y). 

First we are to prove that the isotropy is one-dimensional. Let Z ^ G — I where 
G-{X,Y,i,j) = {X,Y,i,j). ZK[Y]i = and jK[X]Z = imply Zp,q = ifp > t + 2 
or q < t. Therefore Z is upper triangular. We want to prove that Zp^q = unless 
p = I, q = n. We proceed by induction on q — p. 

[Z,X] = [Z,y] = Dimply 



E 

r<m<s 



E 

r<m<s 



Ur^rn 
VmyS 







for every r < s (see [5]). 

For r — t, s = t + 1 we have 



: 



Vt,t 


Zt,t 


+ 


2/t,t+i 












+ 


1 





yt,t+i 


Zt,t+1 




2/t+i,t+i 


Zt+l,t + l 




1 










Zt+l,t+l 
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This proves the case q — p = 0. 



Assume that 



if q — p < u and let r and s be so that s — r = u + 1. We 



are to prove that Zr,s-i = 2^r+i,s = 0. 



= 



= E 

r<m<s 











Xf^f Zf 


r 


+ 










Xf g Zy 


,s 




*^r+l,s ^r-\-l,s 









5-1 


^r,s— 1 










+ 











— Xf^f-^iZf-^i^g Xg 



+ 



+ 



^r,s 



^r, s — 1 



Vs-l.s 

r < t and 5 — 1 > t we have 



Similarly, 





li r > t then r + 1 > i + 2, so z^+i, 
= 



ai 






1 ^r%s-l 


1 






^1 Zr+l^s 



0. 



and 





1 -2^r, s — 1 




1 



r,s— !• 



Similar considerations apply if ,s — 1 <t. 

This proves that Zp^q = Q\ip — q<n — 2 (since in the computations above we 
require r > 1, s <n). Therefore Zp^q = unless p = 1, q = n. 

It is easy to check that if zi.„ is arbitrary and Zp^q = for {p,q) ^ i^ifi) then 
G = I + Z fixes (X, Y, Therefore the isotropy is one- dimensional. 

Now we want to prove that the conjugacy class is uniquely determined by the 
image under \I>. In order to do that we are to find a suitable basis in which X and 
Y are easy to describe. 



-bn 



.i_t_„X"-i-*-'")J>!:'"; < 



Lemma 7. There exists a vector v G V so that 

(a) V ^ ImX + Im Y ; 

(b) ZmV e YK[X,Y]i where = {Y-h^X- 

m < n — 2 — t. 

(c) Zov = {Y -blX 6„_i_tX"-i-*)i; - 0; 

Proof. We will construct v one entry at a time (using the basis from before). 

/ ^1 \ 



Vt+l 
Vt+2 

\vn J 

Let u„ = 1, this guarantees (a). 

Now we prove that Im Z^ C K[X, Y]i. This means that the last n — t rows 

of Zm vanish. Let Y and X be the (n — t) x (n — t) lower right submatrices 
of Y and X respectively; so in fact X is a regular nilpotent matrix, and Y = 
biX + --- + 6„_i_tX"-i-*. Therefore 



(Y - biX bn-l-t-mX 



n—l—t—m 
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It is easy to see that the {t + l)x {t+ 1) upper left submatrix of F — biX 

6„_i_(_mX"~-'^~*~™ is a regular nilpotcnt matrix, in fact, the entries just above 
the main diagonal are all equal to 1 — aibi ^ 0. 

Now consider the t-th row of Zm- The {t + l)-th entry of that row of {Y — 
biX — • • • — bn-i-t-mX'"~^~*~"^ is 1 — oi5i, and all the preceding entries arc equal 
to 0. After muftiplying by x""!-*-™, the {t + 1 + m)-th entry of the t-th row 
of Zm is equal to 1 — aibi, and all the preceding entries are equal to 0. So we 
use Zn-^2-ti Zn-z-t, ■ ■ ■ , Zi to choosc fn-i, 1^)1-2, • • • , vt+2 SO that the t-th entry of 
ZmV is zero, i.e., Z^v e YK[X, Y]i = YK\Y]i = {Yi, YH, YH). 

Since the (t + 1) x {t + 1) upper left submatrix of Zq is regular then, given 
Vt+2, ■ ■ ■ G IK, there are unique numbers V2, ■ ■ ■ , Vt+i G K so that the vector w 
formed in this way is in kei Zq (wi is arbitrary). 

□ 

Since v G V \ ImX + ImY we have that t; is a cyclic vector for X and Y, in 
fact. 









* 




* 




, Xv = 




* 




* 


* 




1 


v) 




^0/ 



X'v 



* 

1 




YX''' 



/A 

1 





..,X"-^-*v- 



/1\ 






Voy 



Voy 



Therefore V = {v,Xv, . . . ,X"-'^-*v,YX'^-'^-*v, . . . ,Y*X'^-'^-*v), K[X,Y]i = 



{YX 



n-l-t. 



,Y'X 



n-l-t 



v), imX,Y])^ = {X' 



*v.YX"- 



Now we write X and Y as matrices in this basis. Since Y acts regularly in 
(jK[X, Y])-^ and X acts as aiY H + at+iF*+^ in this space we have that 



q-p 



l<P<q<t 

q — p=l,t+l<p<n— 1. 
otherwise 



V G ker Zq means 



Yv = biXv 



1 \rn—l — t 

bn-l-tX V 



and for m = 0, . . . , n - 2 - the coefficient of YX"-'^-*v in Y{X"'v) is 0. This 
implies that 



q— p — l,l<p<t 

t +1 <p < q <n 

q — p>0,p = tovq = n 
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For example, for rt = 8, < = 4: 



X 



/o 


ai 


02 


03 


04 












/o 


1 











2/16 


2/17 


0\ 








Ol 


02 


03 



















1 








2/26 


2/27 














Ol 


02 






















1 





2/36 


2/37 

















Ol 

























1 


























1 























bi 


62 


63 




















1 

























bi 


b2 























1 

























bi 


Vo 




















V 




^0 




















0; 



To conclude the proof of the Theorem [4] we use following 

Lemma 8. The entries yp,q for l<p<t, t + 2<q<n, the vector i and the cov- 
ector j are uniquely determined by ai, . . . , at, 61, ... , bn-i-t and by the condition 
rki[X,Y]) = l. 

Proof. We split X and Y in blocks of sizes t, 1, n — 1 — t: 
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etc. 



Then XnYi^ + X12I23 — ^i2-'^^23 — ^13^33 = * j ,a rank one matrix, where i and 
j are the truncated vector and covector respectively. Note that the entry in the 
lower left position of ij is equal to ai6i — 1 7^ 0, and that ^12123 and Yi2^23 are 
determined by ai, . . . , at, 61, ... , bn-i-t- 

Let 



/ii = (1 



0) , /i2 = (0 1 



0) ,, 



(0 



1 



Since htXn = and htYi^ = we conclude that htij = ht (X12I23 — ^i2-'^23) 7^ 
0. From here we get that j is uniquely determined by the parameters ai, . . . , at 
&i, . . . , bn-i^t (up to a constant multiple, that we ignore). 
In general, after we know hrnYi:^ for m > fc , we calculate 

hkij = hk {XnYis + X12Y23 - ^12^23 - >"i3^33) • 

Since hkXn = aihk-i + 02/1^-2 + • • ■ and hkYuX^s = (O 2/fc,t+2 ■ • ■ 2/fe,n-i) 
we conclude that the first entry of hk (X11Y13 + X12F23 — yi2-'i'23 — 5^i3-'^33) does 
not depend on Y13, so neither does hkij. This means that hki depends only on 
ai, . . . , Ot, 61, . . . , 6„_i_t (the key fact here is that ai^i — 1 7^ 0). And with this, 
we can find hkYisX^^ = (O ykj+2 ■ ■ ■ 2/fc,n-i)- Since the last entry of /ife^i3 = 
{yk,t+2 ■ ■ ■ 2/fc,n-i 0) is zero, we are not loosing any information. □ 



This concludes the proof of Theorem ID 



□ 
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From Theorem E] and the fact the "if is dominant we conclude that Nt,n-i-t 
(and therefore 'JIt,n-i-t) is irreducible and its dimension is dim7i|j^^ +dim7-^|p^* + 
dim GL{V) — 1 = n? + n — 2. dim A/q n = dim A/",' ^ = -n? + n — 1 as, was proved in 

ID- 

3. Proof of Theorem [T] 

First we prove the non-redundancy of 77t.n-i-t', 1 < t < n ~ 2. 

Lemma 9. Aft.n-i-t H 77r.s — unless r ~ t and s — n — 1 — t. 

Proof. Since JVr,s C M^^g, then Mt.n-i-t n A/'r,s 7^ implies t < r, n — 1 ~ t < s, so 
n — 1 < r + s, but Afr,s = 0ifr + s>n and r, s > 0, therefore n — 1 = r + s and 
we conclude that t = r, n — 1 — t = s. □ 

To complete the proof of Theorem [1] we have to prove that 

n-2 

t=i 

Theorem 10. IfO<r + s<n-l then TVr,^ C Uo<t<n-i -^t^"-!-*- 

Let {X,Y,i,j) G A/", r = dim]K[X, Let A be as in Lemma|6l The conclusion 
of the Lemma means that A represents a Young diagram of size r. Let A^; = {(a, b) £ 
A|(a + 1, 6) e A}, Xy = {(a, b) € A|(a, 6 + 1) G A}. 

Fix a decomposition V ~ ]K.[X, Y]i(BV' and write X and Y in blocks accordingly: 



^ X2J ' V ^2 

Here and Yi (resp. X3 and Y3) are nilpotent commuting endomorphisms of 
K[X,Y]i (resp. 1/') while X3,Y3 : V K[X,Y]i. We want to describe the subva- 
riety 

AfA-„y„i = {(X'.Y'.f ) £ n X n X F*| 

Lemma 11. Mxi.Yi,i is birationally equivalent to K-n-r x (V'*Y^^ and therefore 
irreducible. 

In concrete terms, we have the following 

Lemma 12. Let 

(a,h)eA (a,fc)eA 

w/iere Q;(a,b), /3(a,fc) G y*. T/ien j', {P(a,b) I (a,^) e A^;} and {q;(o,6) | (0,6) G Xy} are 
uniquely determined by X2, Y2 and the remaining a(a,b)j l^(a,b)- 

Proof. The condition ij' = XiY^ + Y^X!^ - YiX^ - X!^Y^ means 

(1) tf = J2 (^"+'y'*)(/3(a,fc)) + iX^Y^)iP^,^,)X^) - (X«y''+iz)(a(,^,)) 

(a,b)GA 
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= J2 (X"y''^)(/3(,,,)X^-a(„^,)K,')+ E 

(a,b)eA {a,b)£\,a>() 

{a,b)ex,b>a 

where W = E(a,.)eA\A. (^"+'>^'*)(/3(a,6)) - E(a,6)eA\A„ (^">^'+'*)("(a,6))- Note 
that W depends on i, Xi, Yi, \{a,b) G A \ A^;} and {Q;(a {,) | (a, 6) G A \ Aj,}. 

Let W==E(a,6)eA(^"i^'0Ka,6))-^ 

Let {a,b) G A. Consider the appearances of X°-Y''i in equation ([1]). For (a, 6) = 
(0, 0), we have j' — /3(o,o)-''^2 ~ (^{0.0)^2, and for every other (a, 6) G A, 

(2) = /3(a,fc)^2 - "(a, 6)^2 + /3(q-1,&) " "(a.b-l) + 1i'(a,fc) 

where /3(-i,6) = a(a -1) = 0. 
Therefore we can define 

P{a,b} = ~P{a+l,b)X2 + 0-(a+l,b)Y2 + "(a+l.b-l) " W(^a+l,b), («, b) G Xx', 
"(0,b) = /5(0,b+l)-'^2 ~ Q^(0,b+1)^2' + ^^'(0,6+1): (0,6) € A^;. 

This guarantees ^ for every (a, 6) G A \ (0, 0). The definition is non-recursive since 
every a(^a,b) i P{a,b) is defined as a regular function of covectors associated to higher 
monomials in the lex deg order. □ 

A particular case of Lemma [T^ is important in its own right. 

Lemma 13. Let Xi G gt^, I2 G be nilpotent and regular. Then for generic 

a((, ft), /3(a,h) we have that dimj'K[X, y] = rt — 1 — r. 

Proof. Since Xi is regular then Yi = X]u=i with ci,...,Cr-i G IK, z is a 
cyclic vector for Xi and A — {(0, 0), . . . , (r — 1, 0)}. For simplicity we denote at = 
a(t,o), /3t = /3(t,o) so 

1 — 1 r — 1 

ft follows that 

I]' = X^Y^ + yg'x^ - Y^X'^ - X'^Yi 

Cr-1 \ /r-1 \ /r-1 \ /r-1 \ /r-1 \ 

E + E P2- E ^"^i" E - E ^2 

t=0 / \t=0 / \n=l / \t=0 / \t=0 / 

r-1 / t \ 



E XH [l3t-i+ PtX'2 - atY^ - E c««t-. 



t=0 \ u=l / 

where we define /3_i = 0. Then j' = fioX2 — aoY2 and Pt-i + f3tX2 — oitYi ^ 
X]u=i '^"'^t-M ~ f'^'' every t > 1. This implies that ao, . . . , a^-i, G V* 
are arbitrary and /3r-2, ■ • • i/^o are defined recursively by (3t-i = ~/3t^2 + OitYi + 
Z]«=i c«at_„ = 0. 

Now we want to calculate dimj'K[X',y] = dim j'K[X2, 1^2] f^e case where 
Y2 is regular and X2 — X]"=i~ ^1.^2'^- Since /3o = 00^2' + ciao — /3i-'i^2, it turns 
out that i' — (cidi — l)Q;oy2'+terms with higher powers of y2'- It follows that 
dim j'K[X', y'] = n — r — f when cidi — f 7^ and is a cyclic vector for Yj'. □ 
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Proof of Theorem\l(A Let {X,Y,i,j) £ Mr,s- We want to prove that every open 
set U in Af containing {X,Y,i,j) intersects Mt,n-i-t for some t. Since Nxx,Yi,i is 
birationally equivalent to ICn-r x {y'*Y^^ and the pairs of nilpotent matrices where 
one is regular is dense in fCn-r (see U), we can find {X' ,Y' e U C\Mxi,Yi.i 
so that X2 is regular. Now we can reverse the roles of i and j to apply Lemma fT3l 
and find (X", Y" , € C/ C M,„_i^t where t ^ dim j'K[X', Y']. □ 

Remark. The hypothesis about the characteristic of K has only been used for the 
irreducibility of the zero fiber of the Hilbert scheme. In other characteristics it is 
still true that 

n-2 

t=\ 

but the closed sets on the right-hand side may not be irreducible. 

4. rk(A + B) < 1 

Consider the variety 

S = {{A,B,i,j) enxnxVx V*\A + B^ ij}. 

Lemma 14. If {A, B.i, j) G S then A and B can be simultaneously triangularized. 

Proof. We proceed by induction on dimT^, the first case being obvious. Let v G 
keri3\{0}. li jv = then Av = and we can apply induction on V/¥m. Otherwise 
we can assume that jv = I. Then i = ijv — Av + Bv = Av and therefore 
B = ij - A = A{vj - I). Choose w e V* so that wA = 0. Then wB = and 
wi — 0, so we can apply the induction on V* /w. □ 

Recall that K{A, B)i be the smallest subspace of V containing i and invariant 
under A and B. 

Lemma 15. (A + B)|k(a.s)» = 0. 

The proof of this Lemma is essentially the proof of Lemma 2.1.3 in 7J using our 
Lemma [Til 

Proof. We have to prove that {A + B)p{A^ B)i — ijp{A, B)i — for any polynomial 
p in two noncommutative variables. But 

jp{A, B)i = tr(p(A, B)ij) = tr(p(A, B){A + B)) = 

since A and B are upper triangular in the same basis. □ 

As a consequence we can write IK[j4]i = K[A, Bji — K{A, B)i and similarly 
jK[A]=jK[A,B]=jK{A,B). 

Now we want to find the irreducible components of S. Let 

Sr.s = {{A,B,i,j) e S I dimK[A]i < r, dimjK[A] < s}. 

This is clearly a closed set in the Zariski topology. 

Theorem 16. The irreducible components of S 0, I,..., n-1. 
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Proof. Lemma [T4l implies dimK[yl.]i + dimj]K[A] < n~l since, in the basis given by 
the Lemma, the matrix ij is upper triangular and therefore the number of nonzero 
entries of i and j cannot exceed n. Therefore 

n 



r=0 



Let 


















1 

0/ 





1 





(0 



1 



where the I's in v and jr are located in the r-th and (r4 
Then [L, i^jr ~ L, ir, jr) G Sr'^n-r' if and only if r = r' 



l)-th positions, respectively. 
This proves that the closed 



sets Sr 



r — 0, 1, . . . , rt — 1 are non-redundant. 



Now we have to prove that iS^.n-r is irreducible. Let 

Si , = {{A, B, G S I dimK[A]i = r, dim jK[A] = s}. 
We are to prove that 5^ is irreducible and that its closure in the Zariski topology 

is Sr^n—r- 

Let (A, B,i, j) G Sr^n-i~r and define i, ir, jr as before in the same basis that 
is used to triangularize A and B. For r G K consider {A{t), B{t), i{T),j{T)) where 
A{t) = tA+{1- t)L, z(t) = ri + (1 - T)ir, j(r) = rj + (1 - r)>), B{t) = 
i{T)j{T) — A{t). Clearly the curve {(A(t), B(t), i{T),j{T)) \ r G K} is contained in 
Sr,n-r and dimK[yl(r)]i(T) = r, dim j(r)K[yl(T)] — n — r except for finitely many 
values of r. Therefore [A, B, i,j) is in the closure of 5^ 

Every (A,B,i,j) G S!^„_^ can be conjugated into 



A 



A' 
J 



B 



A 



G = 



G'J = JG'. 



where the blocks in A have sizes r and n — r, respectively, and J represents the 
Jordan block of the appropriate size. The elements of the isotropy have the form 

7 G'^ 

The space of r x (n - r) blocks G' with G'J = JG' is isomorphic to ]K"""{'-."-'-} 
and therefore irreducible. We conclude that S'^ is irreducible and its dimension 
is r(n — r) + — min{r, n — r}. 

□ 
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